Now that we have KMS, it happens things go bad enough that when X goes
down, one can no longer access any console to see what
happened. Mainly, one can distinguish between two cases.
- The good one: the machine is still accessible over the
network. Unfortunately, that requires having a spare machine to ssh
into the first machine, but that means one can run
dmesg, look
into X logs, possibly run X inside gdb and see where it breaks
(because glibc s backtrace isn t exactly an excellent tool to
get the idea of what s going on and that s the tool used in X to
print a backtrace in the log through a signal handler).
- The bad one: the machine is no longer accessible over the
network. Likely, the kernel is in such a bad shape that one is
never going to know what happened. Usually, writing logs to disc
didn t happen, so logs won t magically show up at next reboot.
I m going to discuss the second case. There s a nice tool in the
kernel which makes it possible to send kernel logs over the
network. It s called
netconsole. As far as limitations are
concerned, one shall note that it s UDP only, and over Ethernet (in
other words: no wireless). The good news is that it can usually make
the last crucial lines available.
Let s call the crashy machine a
patient and the logging machine a
doctor.
One should keep in mind that if something goes wrong while configuring
the
netconsole module, it can be unloaded at anytime through:
sudo modprobe -r netconsole
Easy case: on a local network
Here s an example: Patient s IP is
192.168.0.1, doctor's IP is
192.168.0.2.
Doctor setup
What one needs on the receiving side: only
netcat. A tiny warning,
there are several implementations. Here s the appropriate syntax:
# netcat-traditional:
nc -l -u -p 6666 tee ~/netconsole.log
# netcat-openbsd:
nc -l -u 6666 tee ~/netconsole.log
The
nc command is handled through the alternatives system, one can
use
update-alternative --config nc to pick the appropriate one if
several
netcat-* packages are installed. To list the current one, an
easy way is
readlink -f $(which nc).
Using
tee means that incoming messages are going to be printed, both
to the standard output and to the specified file.
Running as non-privileged user is sufficient.
Patient setup
Now, to have the patient send stuff to the doctor, a simple
modprobe
is needed:
sudo modprobe netconsole netconsole=@/,6666@192.168.0.2/
What happens here? One requests the
netconsole module to be loaded,
and one specifies the parameters. Details can be read in the Linux
kernel documentation (
Documentation/networking/netconsole.txt), but
concentrating on the points of interest here:
- 192.168.0.2: The doctor s IP.
- 6666: The UDP port to which packets are sent. 6666 is the default
and can be omitted.
That s all! On the doctor side, one should see something along those
lines:
[2920549.188090] console [netcon0] enabled
[2920549.188102] netconsole: network logging started
Slightly harder case: over internet
Because one might not have a second machine handy, it s also possible
to go through a router and send stuff across the internet.
Here s an example: Patient s IP is
192.168.0.1, connecting to
internet through a router, which IP is
192.168.0.254. The doctor,
available over the internet, is
192.0.32.10.
Doctor setup
Exactly the same as in the previous case. The main issue one can run
into is firewalls at this point. That s why one may want to switch
from the default 6666 port to another one which wouldn t be
filtered. One can think of 53 (DNS), but that means running
nc as
root since it s a privileged port (below 1024).
Patient setup
Routing is also possible, but that requires an extra parameter: the
MAC address of the router. To obtain it, one can use the
arp
command:
/usr/sbin/arp 192.168.0.254
Supposing it returned the
01:02:03:04:05:06 address, loading the
module becomes:
sudo modprobe netconsole netconsole=@/,6666@192.0.32.10/01:02:03:04:05:06
Now, if you re running into firewall-related issues, you can change
the source for the UDP packets. The default is 6665, but assuming you
want to send from an unfiltered 1234 port, that becomes:
sudo modprobe netconsole netconsole=1234@/,6666@192.0.32.10/01:02:03:04:05:06
Time for a fifth Debian XSF News issue!
Update: Ulrich mru Dangel suggested pdnsd as nice alternative to dnsmasq and Cyril KiBi Brulebois pointed out, that it s not necessary to invoke dnsmasq after fresh installation as it s running by default then I adjusted the text accordingly, thanks for the pointers!
If /etc/resolv.conf doesn t provide any nameserver entries glibc[1] will automatically initialize the nameserver to the loopback address. This is nice for dealing with chroots without having to manually edit resolv.conf to get working DNS.
Just install a basic DNS forwarder like dnsmasq ( apt-get install dnsmasq ) or if it s already on your box just start it ( /etc/init.d/dnsmasq start ). That s it. Now when chrooting into a system without an existing resolv.conf configuration it will give you a working DNS setup without any further work.
PS: 
so wontfix will do it.
